Federal Motor Carrier Safety: Modifying the Compliance, Safety, Accountability Program Would Improve the Ability to Identify High Risk Carriers
Highlights
What GAO Found
The Federal Motor Carrier Safety Administration's (FMCSA) Compliance, Safety, Accountability (CSA) program has helped the agency contact or investigate more motor carrier companies that own commercial trucks and buses and has provided a range of safety benefits to safety officials, law enforcement, and the industry than the previous approach, SafeStat. Specifically, from fiscal year 2007 to fiscal year 2012, FMCSA more than doubled its number of annual interventions, largely by sending warning letters to riskier carriers.
A key component of CSA--the Safety Measurement System (SMS)--uses carrier performance data collected from roadside inspections or crash investigations to identify high risk carriers for intervention by analyzing relative safety scores in various categories, including Unsafe Driving and Vehicle Maintenance. FMCSA faces at least two challenges in reliably assessing safety risk for the majority of carriers. First, for SMS to be effective in identifying carriers more likely to crash, the violations that FMCSA uses to calculate SMS scores should have a strong predictive relationship with crashes. However, based on GAO's analysis of available information, most regulations used to calculate SMS scores are not violated often enough to strongly associate them with crash risk for individual carriers. Second, most carriers lack sufficient safety performance data to ensure that FMCSA can reliably compare them with other carriers. To produce an SMS score, FMCSA calculates violation rates for each carrier and then compares these rates to other carriers. Most carriers operate few vehicles and are inspected infrequently, providing insufficient information to produce reliable SMS scores. FMCSA acknowledges that violation rates are less precise for carriers with little information, but its methods do not fully address this limitation. For example, FMCSA requires a minimum level of information for a carrier to receive an SMS score; however, this requirement is not strong enough to produce sufficiently reliable scores. As a result, GAO found that FMCSA identified many carriers as high risk that were not later involved in a crash, potentially causing FMCSA to miss opportunities to intervene with carriers that were involved in crashes.
FMCSA's methodology is limited because of insufficient information, which reduces the precision of SMS scores. GAO found that by scoring only carriers with more information, FMCSA could better identify high risk carriers likely to be involved in crashes. This illustrative approach involves trade-offs; it would assign SMS scores to fewer carriers, but these scores would generally be more reliable and thus more useful in targeting FMCSA's scarce resources.
In addition to using SMS scores to prioritize carriers for intervention, FMCSA reports these scores publicly and is considering using a carrier's performance information to determine its fitness to operate. Given the limitations with safety performance information, determining the appropriate amount of information needed to assess a carrier requires consideration of how reliable and precise the scores need to be for the purposes for which they are used. Ultimately, the mission of FMCSA is to reduce crashes, injuries, and fatalities. GAO continues to believe a data-driven, risk-based approach holds promise; however, revising the SMS methodology would help FMCSA better focus intervention resources where they can have the greatest impact on achieving this goal.
Why GAO Did This Study
From 2009 to 2012, large commercial trucks and buses have averaged about 125,000 crashes per year, with about 78,000 injuries and over 4,100 fatalities. In 2010, FMCSA replaced its tool for identifying the riskiest carriers--SafeStat--with the CSA program. CSA is intended to reduce the number of motor carrier crashes by better targeting the highest risk carriers using information from roadside inspections and crash investigations. CSA includes SMS, a data-driven approach for identifying motor carriers at risk of causing a crash.
GAO was directed by the Consolidated Appropriations Act of 2012 to monitor the implementation of CSA. This report examines the effectiveness of the CSA program in assessing safety risk for motor carriers. GAO spoke with FMCSA officials and stakeholders to understand SMS. Using FMCSA's data, GAO replicated FMCSA's method for calculating SMS scores and assessed the effect of changes--such as stronger data-sufficiency standards--on the scores. GAO also evaluated SMS's ability to predict crashes.
Recommendations
GAO recommends that FMCSA revise the SMS methodology to better account for limitations in drawing comparisons of safety performance information across carriers. In addition, determination of a carrier's fitness to operate should account for limitations in available performance information. In response to comments from the Department of Transportation (USDOT), GAO clarified one of the recommendations. USDOT agreed to consider the recommendations.
Recommendations for Executive Action
| Agency Affected | Recommendation | Status |
|---|---|---|
| Department of Transportation | To improve the CSA program, the Secretary of Transportation should direct the FMCSA Administrator to revise the SMS methodology to better account for limitations in drawing comparisons of safety performance information across carriers; in doing so, the Secretary of Transportation should direct the FMCSA Administrator to conduct a formal analysis that specifically identifies: (1) limitations in the data used to calculate SMS scores including variability in the carrier population and the quality and quantity of data available for carrier safety performance assessments, and (2) limitations in the resulting SMS scores including their precision, confidence, and reliability for the purposes for which they are used. |
FMCSA did not agree with this recommendation and has requested that we close the recommendation as not implemented. However, in response to a similar 2017 review by the National Academy of Sciences, FMCSA developed and tested a new methodology to compare safety performance across motor carriers. We believe the changes made could account for the data and other limitations we identified in our report. In February 2023, FMCSA published proposed changes to the SMS methodology for comment in the Federal Register. As of March 2024, FMCSA is still in the process of reviewing comments and has not made a final decision on whether proposed changes would be implemented. To fully implement this recommendation, FMCSA should ensure that the methodology used to compare safety performance across motor carriers addresses the limitations we identified. Without a new methodology, FMCSA's ability to target unsafe motor carriers is hindered by insufficient information not accounted for in the current SMS methodology, such as variability in the motor carrier population. We will continue to monitor FMCSA's efforts to address this recommendation.
|
| Department of Transportation | To improve the CSA program, the Secretary of Transportation should direct the FMCSA Administrator to ensure that any determination of a carrier's fitness to operate properly accounts for limitations we have identified regarding safety performance information. |
The Federal Motor Carrier Safety Administration's (FMCSA) Compliance, Safety, Accountability (CSA) program has helped the agency contact or investigate motor carrier companies that own commercial trucks and buses, providing a range of safety benefits to safety officials, law enforcement, and the industry. A key component of CSA-the Safety Measurement System (SMS)-uses carrier performance data collected from roadside inspections or crash investigations to identify high risk carriers for intervention using safety scores calculated in various categories, including Unsafe Driving and Vehicle Maintenance. FMCSA proposed a rule that would allow the determination of a motor carrier's fitness to operate based on a carrier's safety scores in several of these categories, among other things. In 2014, GAO reported that FMCSA faced two major challenges in reliably assessing safety risk for the majority of carriers using SMS safety scores. First, most regulations that FMCSA used to calculate safety scores are not violated often enough to strongly associate them with crash risk for individual carriers. Second, most carriers lacked sufficient safety performance data to ensure that FMCSA could reliably compare them with other carriers. FMCSA officials told GAO that the primary purpose of SMS is to serve as a general radar screen for prioritizing interventions. However, GAO found that due to insufficient data SMS is not as effective as it could be for this purpose. Further, if the same safety performance data used to inform SMS scores are intended to help determine a carrier's fitness to operate, most of these same limitations will apply. As a result, GAO recommended that FMCSA ensure any determination of a carrier's fitness to operate properly account for limitations we identified regarding safety performance information. In 2020, GAO confirmed that FMCSA withdrew the proposed rule to allow the use of SMS safety scores in safety fitness determinations. Furthermore, FMCSA does not have any plans to use safety scores to determine motor carrier's fitness to operate in the future. By taking this action, FMCSA lessened the risk of using unreliable safety performance information when determining a motor carrier's fitness to operate safely on the nation's highways.
|